Exploiting heterogeneity in operational neural networks by synaptic plasticity

Abstract The recently proposed network model, Operational Neural Networks (ONNs), can generalize the conventional Convolutional (CNNs) that are homogenous only with a linear neuron model. As heterogenous ONNs based on generalized model encapsulate any set of non-linear operators to boost diversity and learn highly complex multi-modal functions or spaces minimal complexity training data. However, default search method find optimal in ONNs, so-called Greedy Iterative Search (GIS) method, usually takes several sessions single operator per layer. This is not computationally demanding, also heterogeneity limited since same will then be used for all neurons each To address this deficiency exploit superior level heterogeneity, study focus drawn searching best-possible set(s) hidden “Synaptic Plasticity” paradigm poses essential learning theory biological neurons. During training, library evaluated by their synaptic plasticity level, ranked from worst best, an “elite” ONN configured using top-ranked sets found at Experimental results over challenging problems demonstrate elite even few layers achieve performance than GIS-based as result, gap CNNs further widens.